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SECTION  I 

INTRODUCTION 


The  Navy  is  cautiously  expanding  the  use  of  computer-aided 
instruction  (CAI)  in  classrocms  and  enbedded  training  in  operational 
environments.  If  the  effectiveness  of  such  instruction  is  to  be 
maximized,  attention  must  be  given  to  the  user-corrputer  interface  (UCI). 
The  ease  with  which  students  ccmrunicate  with  the  computer,  in  part,  will 
determine  hew  well  they  learn.  In  the  case  of  enbedded  training ,  the  UCI 
may  determine  whether  the  training  is  used  at  all. 

Learning  conplex  jdb  tasks  consumes  students'  attention  and  memory 
resources.  Additional  demands  sinply  to  understand  the  complex  system 
serving  as  the  medium  of  instruction  should  be  minimized.  Design  of  CAI 
according  to  human  factors  principles  will  lead  to  a  good  match  between 
cognitive  processes  of  students  and  the  information-processing 
capabilities  of  computers.  The  result  will  be  instruction  which  uses  all 
the  advantages  of  the  computer  as  a  training  medium  to  produce  high 
information  transfer,  reduced  mental  work  load,  reduced  human  error  and 
efficient  learning. 

Research  articles  and  guidelines  on  the  design  of  CAI  and  on  the  UCI 
have  proliferated  in  recent  years.  These  studies  and  guidelines  have  not 
been  organized  and  evaluated  for  their  relevance  to  training 
effectiveness. 

The  Naval  Training  Systems  Center  is  conducting  research  to  produce 
human  factors  guidelines  for  the  design  and  evaluation  of  computer-based 
training  systems.  The  first  product  of  this  research  effort,  reported 
here,  is  a  set  of  draft  guidelines,  with  accaipanying  references,  for  the 
design  of  CAI.  These  guidelines  were  derived  from  literature  reviews, 
existing  guidelines  and  standards,  and  empirical  and  theoretical  research 
pertaining  to  CAI  and  the  UCI.  Special  attention  has  been  given  to 
information-processing  theory  which  focuses  on  the  cognitive  demands 
placed  upon  an  operator  of  a  computer  system. 

The  guidelines  presented  in  this  report  are  organized  according  to  a 
set  of  human-conputer  dialogue  principles  proposed  by  Williges  and 
Williges  (1984)  and  are  formatted  as  the  CAI  Evaluation  Checklist.  The 
checklist  format  was  chosen  to  examine  its  potential  for  evaluating  the 
interaction  of  a  student  with  a  computer.  Additional  research  will  be 
conducted  to  validate  the  instrument's  ability  to  predict  the  training 
effectiveness  of  CAI.  Future  research  will  examine  the  checklist's 
ability  to  measure  the  quality  of  CAI  courseware  and  hardware,  in  terms  of 
how  well  a  system  displays  information,  hew  easy  it  is  for  students  to 
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operate  a  system,  and  how  well  the  CAI  capitalizes  on  the  advantages  of 
the  computer  as  a  training  medium. 


PURPOSE 

This  report  provides  guidelines  in  checklist  forrot  for  the  design  of 
the  user-ccrtputer  interface  (UCI)  for  computer-assisted  instruction  (CAI). 
As  such,  it  will  be  of  interest  to  human  factors  professionals, 
instructional  technologists  and  engineers  who  design  or  develop 
conputer-based  training  systems  or  authoring  systems  for  CAI. 

The  checklist  format  is  useful  for  evaluating  CAI  as  part  of  the 
final  acceptance  process  for  the  government  acquisition  of  CAI  systems  and 
also  to  those  doing  research  on  CAI  or  the  UCI  who  need  quantitative 
methods  of  evaluation. 
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SECTION  II 

DEVELOPMENT  OF  THE  CHECKLIST 


The  guidelines  were  produced  from  a  review  of  the  behavioral  research 
literature,  existing  guidelines,  and  verbal  reports  of  experienced  CAI 
developers.  Information  on  the  design  of  CAI  was  ccrbined  with  studies  of 
the  UCI  in  an  effort  to  merge  the  two  areas.  Appendix  A  includes  the 
guidelines  and  specific  references  from  which  they  were  derived. 


BASIC  PRINCIPLES  OF  GOOD  COMPUTER  SYSTEM  DESIGN 

The  Williges  and  Williges  (1984)  report  on  human  factors 
consideration  in  the  design  of  human-conputer  dialogue  provided  an 
organizational  structure  for  the  material.  All  guidelines  were  organized 
using  five  of  the  Williges'  principles:  brevity,  consistency, 
flexibility,  compatibility,  and  responsiveness.  Fran  the  Williges'  (1984) 
review  of  existing  standards  and  guidelines  on  the  human-computer 
interface,  they  concluded  that  these  principles  are  fundamental  to  "good 
computer  system  design."  The  definition  of  each  principle  was  modified  to 
fit  the  requirements  of  CAI  as  a  particular  form  of  human-corrputer 
interaction.  The  modified  principles  are  described  below. 

BREVITY 

Information  a  student  must  maintain  in  short-term  memory  or  attend  to 
should  be  minimized.  Methods  of  chunking  and  focusing  information  aid  in 
reducing  mental  workload.  Early  research  on  information  processing  in 
humans  supports  this  principle  (Miller,  1960;  Norman,  1976).  Humans  have 
a  limited  capacity  to  process  quantities  of  information  at  one  time  and 
therefore  nust  selectively  attend  or  use  other  control  processes  to 
receive  and  remenber  information. 

CONSISTENCY 

Task  demands  must  be  consistent  within  a  training  system  to  develop 
user  expectations.  If  a  student  develops  a  correct  mental  model  of  the 
system,  there  will  be  a  dramatic  reduction  in  cognitive  processing 
expended  on  understanding  the  computer  and  hew  it  works. 
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Recent  studies  invoke  the  concept  of  mental  models  to  explain  skilled 
interaction  with  a  system.  These  models  explain  learning  as  the 
development  of  successive  mental  representations  of  the  system  and  the 
acquisition  of  a  set  of  rules  for  interacting  with  that  system  (Card, 
Moran,  and  Newell  (1980).  With  repetition  of  particular  ccrrbinations  of 
actions,  cognitive  processing  becomes  proceduralized,  or  automatic 
(Anderson,  1981). 

Theories  on  the  deve lopment  of  automatic  skills  stress  the  importance 
of  training  the  consistent  ccrponents  of  the  training  tasks  (Schneider  and 
Shiffrin,  1977). 

FLEXIBILITY 

A  system,  including  the  instruction,  must  accommodate  individual 
differences  among  students.  A  ccnputer  system  must  have  the  capabilities 
for  review  and  branching,  and  the  instruction  should  include  skill 
diagnosis  and  remediation.  Flexibility  also  includes  capabilities  for 
student  control  over  the  instruction,  such  as  pacing. 

Individual  differences  is  a  central  concern  in  human  learning  and 
performance.  The  cognitive  processes  (e.g.,  memory  operations  such  as 
storing,  searching  and  retrieving)  that  intervene  during  learning  and 
performance  are  individual  difference  variables  to  be  considered  in 
designing  adaptive  instruction  (Glaser  &  Resnick,  1972;  Hunt  &  Lansman, 
1975). 

COMPATIBILITY 

To  minimize  the  information  processing  between  stimulus  and  response, 
input  and  output  formats  must  be  ccnpatible  with  each  other  and  with 
established  behavior  patterns  of  the  students  using  the  system.  There 
must  be  a  minimum  of  translation,  decoding,  and  other  forms  of  cognitive 
processing  necessary  to  understand  conputer  output  of  information  and  to 
know  what  response  (input)  is  required. 

First,  hew  information  is  presented  must  be  compatible  with  the 
required  format  for  student  responses.  When  tasks  are  designed  such  that 
the  response  mode  is  cognitively  ccnpatible  with  the  presentation  mode, 
performance  is  faster  and  more  accurate  than  when  inccrpatibility  exists. 
Stimulus-Central  processing -Response  (S-C-R)  Carpatibility  theory  and 
related  enpirical  research  support  these  contentions  (Wickens,  Sandry,  & 
Vidulich,  1983).  The  theory  maintains  that  when  a  stinulus  is  presented, 
central  processing  of  that  stimulus  will  either  be  verbal  or  spatial  in 
nature.  Verbal  responses  are  cognitively  compatible  with  verbal 
processing  tasks  and  motor  responses  with  spatial  processing  tasks. 
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Second,  input  and  output  trust  conform  to  population  stereotypes, 
e.g.,  red=stop,  green=^go.  If  cues  used  in  the  training  material  must  be 
encoded  in  memory  such  that  they  conflict  with  previous  encoding  of  those 
cues,  the  result  will  be  increased  information  processing  (interference) 
during  retrieval  frcm  memory  (Tulving,  1976). 

RESPONSIVENESS 

The  system  must  provide  informative  feedback  about  student 
performance  and  system  condition.  Optimal  timing  of  system  responses  to 
student  input  will  help  the  students  to  know  where  they  are,  what  they 
have  done,  whether  or  not  it  was  successful,  and  hew  to  take  corrective 
action  as  necessary. 

Research  indicates  that  feedback  which  provides  information,  not 
sinply  immediate  feedback,  is  the  key  to  performance  change  (Cohen,  1985). 
Knowledge  of  results  in  the  form  of  immediate  feedback  like  "CORRECT"  or 
"ERROR,"  are  often  not  enough  to  correct  the  problem.  Informational 
feedjack  helps  the  student  locate  the  error  and  construct  an  alternative 
response,  e.g.,  "Ihe  first  part  of  your  answer  is  not  valid,  check  your 
arithmetic." 


ORGANIZATION  OF  THE  GUIDELINES 

The  authors  used  the  five  principles  described  dbove  to  sort  50 
guidelines  into  five  categories,  and  then  dotained  expert  opinion  on  the 
assignment  of  guidelines  to  the  categories.  The  categorized  checklist 
items  were  distributed  to  one  specialist  each  in  computer  software, 
education,  and  human  factors  for  their  review  and  cements.  They  provided 
constructive  coments  on  the  clarity  and  inporta  nee  of  items  and  their 
appropriateness  to  a  given  category.  Ihe  author  used  these  cements  to 
make  appropriate  modifications  to  the  checklist. 

The  unique  categorization  opened  up  possibilities  of  measuring  the 
quality  of  the  UCI  through  the  measurement  of  five  characteristics — 
brevity,  consistency,  flexibility,  carpatibility ,  and  responsiveness.  A 
sinplified  scoring  method  was  derived  to  beg  in  a  study  of  these 
possibilities. 


SCORING  METHOD 

Ihe  nunber  of  guidelines  in  each  category  was  kept  to  a  maximum  of 
ten  items.  A  yes/no  format  was  used  to  make  the  checklist  easy  to  use. 
The  user  is  asked  to  mark  "yes"  if  the  system  contains  the  feature  for  the 
most  part,  and  "no"  if  the  system  rarely  or  never  incorporates  this 
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feature.  The  "n/a"  response  was  added  for  those  cases  where  an  item  does 
not  apply.  The  items  were  given  equal  weighting  since  a  weighting  method 
is  not  yet  developed.  A  score  in  each  category  can  be  obtained  by  the 
following  formula: 

Score  =  Nunber  of  "yes"  responses 

-  X  100% 

Total  nunber  of  items  -  "n/a"  responses 


This  scoring  method  results  in  scores  for  each  category  ranging  fran 
0%  to  100%.  Criterion  scores  for  acceptable  design  of  computer-aided 
instruction,  currently  unknown,  will  be  developed  and  refined  through 
empirical  research. 


RATER  RELIABILITY 

In  an  initial  attempt  to  obtain  rater  reliability,  education  and 
psychology  professionals  were  asked  to  use  the  CAI  Evaluation  Checklist  to 
evaluate  two  articles  of  interactive  video  courseware:  Trace-A-Ground  and 
AC  Switchboard  developed  for  the  Electrician's  Mate  Navy  "A"  school.  The 
content  was  introductory  material  for  electronics  training. 

Four  evaluators  viewed  the  Trace-A-Ground  courseware  and  five 
evaluators  viewed  the  AC  Switchboard  material.  Rater  reliability  was 
assessed  using  the  analysis  of  variance  approach  to  correlation  (Winer, 
1962).  The  scores  on  each  of  the  five  principles  were  set  up  for  a  2  x  2 
ANOVA  with  the  principles  as  one  factor  and  judges  as  the  other  factor. 

TWo  analyses  were  performed,  one  for  each  set  of  materials.  The  obtained 
mean  squares  were  sibstituted  into  the  Spearman-Brown  prophecy  formula  to 
obtain  estimated  reliability  of  the  averaged  scores  on  the  five  principles 
for  two,  three,  four,  and  five  raters.  The  first  ANOVA  showed  that  the 
reliability  of  the  average  of  the  four  raters  who  evaluated  Trace-A-Ground 
was  r=.87.  This  means  that  am  average  score  on  brevity,  consistency, 
flexibility,  ccnpatibility,  and  responsiveness  from  four  raters  will 
correlate  r=.87  with  the  averages  obtained  from  a  second  set  of  four 
raters.  The  second  ANOVA  showed  the  reliability  of  the  average  of  the 
five  raters  who  evaluated  AC  Switchboaird  to  be  r=.91.  The  Spearman-Brown 
prophesy  formula  indicated  that  acceptable  reliability  (.80)  can  be 
achieved  with  only  two  raters.  Further  information  can  be  obtained  frcm 
Pfeiffer,  Miller,  Platt,  Green,  Monroe,  amd  Traxler  (1986). 
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AUTOMATED  CHECKLIST 

For  the  purposes  of  simplifying  data  collection,  the  CAI  Evaluation 
Checklist  has  been  programmed  in  BASIC  on  a  TRS-80  Model  200  lap-top 
portable  ccnputer.  The  ccnputer  weighs  about  4  lbs.  and  can  be 
battery-operated.  The  machine  uses  a  40  x  16  character  LCD  screen  and  has 
a  full-size  "CWEKTY"  keyboard.  Data  storage  is  in  the  72k  byte  RAM  and 
additional  portable  disk  drives  are  available.  Programs  are  available  to 
interface  the  Model  200  with  other  microcomputers  so  that  files  can  be 
written  directly  to  disk. 

The  automated  checklist  is  useful  for  large-scale  data  collection  and 
data  analysis.  Data  are  stored  for  each  rater  and  the  averages  of  all 
raters  are  computed.  Plans  are  to  program  computations  for  obtaining 
reliability  scores,  and  to  program  diagnostic  feedback .  As  an  example, 
the  diagnostic  feedback  will  contain  information  on  how  best  to  revise  the 
materials  considering  what  changes  should  have  the  highest  priority. 
Prioritizing  will  be  possible  when  further  research  proxluces  a  method  for 
weighting  individual  checklist  items. 
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SECTION  III 

CONCLUSIONS  AND  RESEARCH  RECOMMENDATIONS 


The  goal  of  subsequent  research  in  this  project  is  to  provide  human 
factors  professionals,  engineers,  and  education  specialists  with  a  valid 
and  reliable  checklist  for  computer-assisted  instruction  evaluation.  An 
additional  effort  could  produce  military  specifications  from  the  checklist 
items. 

The  following  areas  will  receive  attention  in  continuing  research  to 
determine  the  validity  of  the  checklist: 

1)  Mdition  of  guidelines  to  the  item  pool,  and  determination  of  the 
appropriateness  of  guidelines  to  a  category, 

2)  Improvement  of  the  scoring  method  by  the  weighting  of  items  in  each 
category , 

3)  Interpretation  of  scores  and  diagnostic  feedback  through  automation. 

4)  Empirical  tests  of  the  checklist's  ability  to  predict  training 
effectiveness. 


CHECKLIST  IMPROVEMENTS 

Further  review  of  the  literature  and  on-going  research  will  add  to 
the  data  base  on  the  human -computer  interface  and  ccrputer-assisted 
instruction.  The  reviews  may  produce  additional  items  for  the  checklist. 

To  determine  the  categorization  of  items,  human  factors  experts, 
software  engineers,  avid  education  specialists  will  be  asked  to  categorize 
the  items  using  the  five  dimensions  described  earlier  (Williges  and 
Williges,  1984).  Items  will  be  shifted  to  another  category  if  there 

is  high  agreement  among  the  experts  that  a  change  is  warranted.  Items  not 
reliably  assigned  to  a  category  will  be  revised  or  dropped  from  the 
checklist. 

A  new  method  of  scoring  based  on  weighted  items  will  be  developed. 
After  items  have  been  categorized,  the  Likert  scaling  technique  will  be 
used  to  weight  each  item  within  the  categories.  Experts  will  be  asked  to 
rate  each  item  on  its  importance  for  effective  learning.  The  weighted 
scoring  method  will  increase  the  capability  of  the  checklist  to  predict 
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training  effectiveness,  and  will  contribute  to  more  accurate  diagnostic 
feedback . 

The  Likert-scaling  format  will  replace  the  yes-no  format  of  the 
checklist  if  research  indicates  inprovement  in  the  instrument's  validity. 


DIAGNOSTIC  FEEDBACK 

Weighted  checklist  items  will  prioritize  for  the  user  those  items 
most  significantly  related  to  training  effectiveness.  If  scores  are 
entered  on  the  automated  version  of  the  checklist,  it  will  be  possible  to 
otain  automated  diagnostic  feedback.  This  feedback  will  interpret  each  of 
the  five  scores  in  terms  of  what  revisions  are  most  crucial  for  increasing 
the  quality  of  the  CAI. 


EMPIRICAL  VALIDATION 

Empirical  research  will  be  conducted  to  evaluate  the  potential  of  the 
CAI  Evaluation  Checklist  to  predict  training  effectiveness.  Over  50 
government  agencies  and  private  firms  involved  in  the  development  or 
evaluation  of  canputer-assisted  instruction  have  been  given  a  copy  of  the 
CAI  Evaluation  Checklist.  Data  will  be  dotained  on  the  applicability  of 
the  checklist  to  various  types  of  CAI  (e.g.,  tutorial,  drill  and  practice, 
simulation,  games).  If  measures  of  student  performance  are  available,  or 
other  evaluative  measures,  they  will  be  reported  along  with  checklist 
measures . 

The  next  major  research  effort  will  focus  on  experimental  designs  to 
test  the  ability  of  the  checklist  to  predict  training  effectiveness.  One 
research  plan  is  to  ccrpare  two  versions  of  CAI  with  the  same 
instructional  content,  one  designed  with  the  checklist  guidelines  and  one 
without.  The  two  CAI  versions  will  be  rated  independently  with  the 
checklist  after  all  the  design  and  development  have  been  completed.  The 
two  CAI  versions  will  then  be  ccnpared  on  training  effectiveness. 

Using  a  between  subjects  design,  students  will  learn  one  of  the  two 
versions  and  statistical  techniques  will  be  used  to  correlate  student 
performance  with  the  CAI  Evaluation  Checklist  measures. 


MILITARY  SPECIFICATIONS 

Another  research  effort  is  to  rewrite  the  design  guidelines  in  the 
form  of  military  specifications.  A  team  of  education  specialists  involved 
in  front-end  analysis  and  human  factors  experts  will  use  the  checklist 
items  as  the  basis  for  a  Data  Item  Description  (DID).  The  DID  could 
accompany  contractual  agreements  with  the  Naval  Training  Systems  Center 


9 


NAVTRASYSCEN  TR86-O02 


vrtien  the  item  to  be  delivered  is  ccnputer-based  instruction  or  interactive 
videodisk  instruction. 


CONCLUSIONS 

The  CAI  Evaluation  Checklist  has  been  derived  from  research  and 
guidelines  on  the  design  of  the  human-computer  interface  for 
computer-aided  instruction.  The  guidelines  have  been  formatted  as  a 
checklist  to  be  used  as  an  evaluation  tool.  Proposed  research  should 
improve  the  validity  of  the  checklist  and  improve  the  scoring  procedure. 

A  checklist  that  is  easy  to  administer,  has  high  face  validity,  and 
is  correlated  with  real  world  performance  should  gain  wide  acceptance  as 
an  extremely  cost-effective  evaluation  tool. 

It  is  proposed  that  the  checklist  items  be  rewritten  as  military 
specifications  for  computer-based  instruction  delivered  to  the  Naval 
Training  Systems  Center. 
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Table  1.  BREVITY  items  and  associated  references. 


M 


BREVITY  ITEM 

Large  portions  of  text  are  broken 
into  meaningful  "chunks". 

No  more  than  seven  lines  of  text 
per  screen. 

Graphic  displays  activate  fran  15% 
to  25%  of  the  screen  area. 

Main  menus  and  sub -menus  have  3  to 
9  choices. 

Use  of  color,  boxing,  and  highlighting, 
rather  than  blinking  to  focus  atten¬ 
tion  on  inportant  segments  of  text. 

No  more  than  3  or  4  text  screens 
without  interactivity. 

No  more  than  10%  of  the  screen  display 
is  highlighted  at  one  time. 

The  time  required  for  a  typical  session 
(or  lesson)  is  within  the  attention 
span  of  the  target  audience. 

Sentences  have  simple  syntax:  active 
voice,  not  conpounded. 


REFERENCES 

Caldwell,  1980;  Kearsley  and 
Hillelsohn,  1982 

Gilmore,  1985;  Karlsrud,  1985 


TUllis,  1983 


Lee  and  McGregor,  1985;  Tijerina, 
Chevalaz,  and  Meyers,  1985 

Caldwell,  1980;  Williges  and 
Williges,  1984;  Belezza  and 
Cheney,  1973 

Mahoney  and  Lyday,  1984 


Gilmore,  1985 


Kearsley  and  Hillelsohn,  1982 


Boyd  and  Eldridge,  1984;  Glynn 
and  Britton,  1984 
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Table  2.  CONSISTENCY  items  and  associated  references. 


CONSISTENCY  ITEM  REFERENCES 

Functionally  alike  screens  are  formatted  Sawyer,  1985;  Williges  and 
in  the  same  way.  Williges,  1984 

When  functional  areas  are  erased,  Heines,  1984 

they  are  consistently  rewritten 
in  the  same  order. 

Consistent  use  of  labels  and  graphics  Gilmore,  1985;  Military  Standard 
keeps  the  same  types  of  frames  MIL-STD-1472C,  1981 

identified  as  such. 

Critical  information  is  always  presented  Gilmore,  1985;  Swezey  and  Davis 
at  the  beginning  of  a  message  or  1983 

centered  on  the  screen. 

Students  receive  constant  delay  of  Kearsley  and  Hillelsohn,  1982; 

feedback  (no  more  than  2  seconds)  McCaan,  1983 

rather  than  variable  delays. 

Consistency  in  the  way  questions  are  Sawyer,  1985;  Williges  and 

asked  and  the  required  format  for  Williges,  1984 

responses . 

The  structure  of  the  presentation  is  McCaan,  1983;  Snyder,  Happ, 

evident  to  the  user  through  the  use  Male us,  Paap,  and  Lewis,  1985 

of  menus  and  concepts  maps. 

A  synbol  has  the  same  meaning  at  all  Gilmore,  1985 

times. 
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Table  3.  FLEXIBILITY  items  and  associated  references. 


FLEXIBILITY  ITEM 

A  page-back  capability  allows  the 
student  to  review  previous 
material. 

Students  can  easily  exit  lessons, 
return  to  menus,  and  exit  the 
prog  ram. 

Student  has  control  over  the  rate 
of  presentation  of  frames. 

Flexibility  in  recognizing  correct 
responses  (e.g.,  misspellings, 
partial  answers). 

The  student  can  request  more  lengthy 
messages,  through  helps,  if  furthur 
clarification  is  needed. 


REFERENCES 
Caldwell,  1980 


Caldwell,  1980;  Mahoney  and 
Lyday,  1984 

Caldwell,  1980 


Caldwell,  1980;  Kearsley  and 
Hillelsohn,  1982;  McPherson- 
Tumer,  1979 

Gilmore,  1985 


The  program  contains  activities  for  Caldwell,  1980 

diagnosis  of  skills  already  mastered. 

There  are  remedial  exercises  for  skill  Caldwell,  1980 
deficiencies. 


Modularized  program  (with  menus)  allows  Caldwell,  1980;  Sawyer,  1985 
the  student  to  begin  at  a  point 
appropriate  to  past  achievement. 

The  student  can  choose  the  difficulty  Boyd  and  Eldndge,  1984; 

level  of  problems  or  exercises  Caldwell,  1980;  Sawyer,  1985 

(achieved  by  variation  in  the  use  of 
prompts ) . 

The  student  can  correct  an  input  error 
(e.g.,  with  BACKSPACE)  or  recover  frcm 
input  errors  without  disrupting  the 
lesson  sequence. 


Williges  and  Williges,  1984 
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Table  4.  COMPATIBILITY  items  and  associated  references. 


COMPATIBILITY  ITEM  REFERENCES 

The  response  mode  is  appropriate  to  Williges  and  Williges,  1984 

the  target  audience. 

Students  are  required  to  use  codes  Caldwell,  1980 

for  responding  only  when  necessary, 

as  in  multiple  choice  answering 

(e.g.,  l=yes,  2=no  is  unnecessary 

coding ). 

Visual  information  and  visual  tasks  Wickens,  Sandry,  and  Vidulich, 

such  as  locating  or  repositioning  1983 

are  presented  graphically. 

Vtiere  frames  are  labeled,  title  not  Boyd  and  Eldridge,  1984 

nunber  is  used  for  identification. 

If  commands  are  entered  by  keyboard,  Williges  and  Williges,  1984; 

the  student  types  the  first  letter  Shinar,  Stem,  BUbis,  and 

rather  than  numeric  code  (e.g . ,h=help) .  Ingram,  1985 

Input,  output  is  consistent  with  user  Barhard,  Hammond,  Morton,  and 

population  stereotypes  (e.g.,  correct  Long,  1981 

(e.g.,  correct  response  feedback  in 
green) . 

Vfriere  order  of  lessons  is  inportant,  Tijerina,  et  al.,  1985 

menu  options  are  listed  by  nunber 
not  by  letter. 

To  clarify  drill  or  test  instructions,  Caldwell,  1980 
a  sample  item  is  answered  before  the 
drill  or  quiz  begins. 

A  response  is  demanded  while  the  Caldwell,  1980 

instructions  on  hew  to  respond  are 
still  on  the  screen. 

Routing  menus  are  limited  to  a  maximum  Heines,  1984 
of  three  levels. 


19 


NAVTRASYSCEN  TR86-002 


Table  5.  RESPONSIVENESS  items  and  associated  references. 


RESPONSIVENESS  ITEM  REFERENCES 

Viien  the  student  must  stand  by.  Military  Standard  MIL-STD- 

periodic  feedback  indicates  1472C,  1981;  Williges  and 

normal  operation.  Williges, 1984 

The  computer  tracks  response  patterns  Cohen,  1985;  Mahoney  and 
and  lists  areas  where  the  student  Lyday,  1984 

remediation. 

Feedback  and  directions  are  clearly  Williges  and  Williges,  1984 

distinguishable  from  other  text  through 
use  of  color,  boxing,  reverse  video,  etc. 

Students  can  dbtain  a  score. 

At  higher  mastery  levels,  students 
are  given  immediate  knowledge  of  right 
and  wrong  responses,  and  more  lengthy 
feedback  is  delayed  until  the  end  of 
the  session. 

There  is  a  pause  after  feecback, 
before  the  lesson  continues,  to 
allow  time  for  consolidation  of 
the  newly  acquired  material. 

Access  to  helps,  references,  or  Gilmore,  1985 

resources  is  easily  available. 

At  the  beginning  of  training,  feecback  Caldwell,  1980 

is  response  informative  (e.g.,  the  - 

part  of  your  answer  is  incorrect.") 

The  student  gets  more  than  one  chance  Caldwell,  1980;  Military 

to  give  the  answer  (with  prompts ).  Standard  MIL-STD-1472C,  1981 

It  takes  no  more  than  5  seconds  for  Gilmore,  1985;  Kearsley  and 

text  and  graphics  to  fill  the  screen.  Hillelschn,  1982 


Kearsley  and  Hillelsohn,  1982; 
Ramsey  and  Atwood,  1979 


Cohen,  1985;  McFherson- 
Tumer,  1979 

Cohen,  1985 
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CAI  EVALUATION  CHECKLIST 


The  CAI  Evaluation  Checklist  examines  the  user-ccrnputer  interaction 
involved  in  computer  aided  instruction.  The  checklist  is  based  on  five 
human  factors  design  guidelines  for  computer  systems  proposed  by  Williges 
and  Will iges  (1984):  brevity,  consistency,  flexibility,  responsiveness 
and  compatibility. 

High  scores  on  these  factors  purport  to  lead  to  high  information 
transfer,  low  mental  workload,  and  reduction  of  human  error. 
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If  you  use  this  checklist  to  evaluate  CAI  courseware,  please  return  the 
following  information  to  me. 

1.  Name  of  your  organization  _ 

2.  Name  of  CAI  course  or  lesson  _ 

3.  Type  of  CAI  (You  may  check  more  than  one.) 

Drill  and  practice  _  Game  _ 

Tutorial  _  Intelligent  CAI  _ 

Simulation  _  Embedded  Training  _ 

4.  Rate  the  courseware  based  on  your  own  expert  opinion: 

poor  average  very  good 

5.  Record  the  scores  you  obtained  from  the  checklist: 

yes  no  n/a 

Brevity  _  _  _ 

Consistency  _  _  _ 

Flexibility  _  _  _ 

Compatibility  _  _  _ 

Responsiveness  _  _  _ 

Return  this  page  to: 

Cheryl  J.  Hamel,  Ph.D. 

Human  Factors  Division,  Code  711 
Naval  Training  Systems  Center 
Orlando,  FL  32813 
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CONSISTENCY 


yes  no  n/a 


In  a  consistent  system,  the  user 
will  develop  a  conceptual  model  of 
how  the  system  works  and  will  gain 
conf idence . 


1.  Functionally  alike  screens  are  formatted  in  the  same  way. 

2.  When  functional  areas  are  erased,  they  are  consistently 
rewritten  in  the  same  order. 


i 


3.  Consistent  use  of  labels  and  graphics  keeps  the  same  types 
of  frames  identified  as  such. 

4.  Critical  information  is  always  presented  at  the  beginning 
of  a  message  or  centered  on  the  screen. 

5.  Students  receive  constant  delay  of  feedback  (no  more  than 
2  seconds)  rather  than  variable  delays. 

6.  Consistency  in  the  way  questions  are  asked  and  the  required 
format  for  responses. 

7.  The  structure  of  the  presentation  is  evident  to  the  user 
through  the  use  of  menus  and  concepts  maps. 

8.  A  synbol  has  the  same  meaning  at  all  times. 


.w 
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COMPATIBILITY 

The  input-output  format  focuses 
attention  on  the  task  to  be  learned, 
not  on  the  task  of  learning  the  system. 


yes  no  n/a 


1.  The  response  mode  is  appropriate  to  the  target  audience. 

2.  Students  are  required  to  use  codes  for  responding  only  when 
necessary,  as  in  multiple  choice  answering  (e.g.,  l=yes, 

2=no  is  unnecessary  coding ) . 

3.  Visual  information  and  visual  tasks  such  as  locating  or 
repositioning  are  presented  graphically. 

4.  Vhere  frames  are  labeled,  title  not  number  is  used  for 
identification. 

5.  If  commands  are  entered  by  keyboard,  the  student  types  the 
first  letter  rather  than  numeric  code  (e.g. ,h=help). 

6.  Input,  output  is  consistent  with  user  population 
stereotypes  (e.g.,  correct  response  feedback  in  green). 

7.  tt\ere  order  of  lessons  is  important,  menu  options  are  listed 
by  nunber,  not  by  letter. 

8.  To  clarify  drill  or  test  instructions,  a  sanple  item  is 
answered  before  the  drill  or  quiz  begins. 

9.  A  response  is  demanded  while  the  instructions  on  hew  to 
respond  are  still  on  the  screen. 

10.  Pouting  menus  are  limited  to  a  maximum  of  three  levels. 
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BREVITY 

Displays  are  simple  and  well-organized 
for  high  information  transfer. 

yes  no  n/a 

—  -  1.  Large  portions  of  text  are  broken  into  meaningful  "chunks". 

—  2.  No  more  than  seven  lines  of  text  per  screen. 

—  -  3.  Graphic  displays  activate  from  15%  to  25%  of  the  screen  area. 

—  -  4.  Main  menus  and  sdb -menus  have  3  to  9  choices. 

—  -  5.  Use  of  color,  boxing,  and  highlighting,  rather  than  blinking , 

to  focus  attention  on  inportant  segments  of  text. 

—  —  —  6.  No  more  than  3  or  4  text  screens  without  interactivity. 

—  7.  No  more  than  10%  of  the  screen  display  is  highlighted 

at  one  time. 

—  -  8.  The  time  required  for  a  typical  session  (or  lesson)  is  within 

the  attention  span  of  the  target  audience. 


9.  Sentences  have  simple  syntax:  active  voice,  not  ccrpounded. 


NAVTRASYSCEN  TR86-002 


FLEXIBILITY 

The  program  adapts  to  individual 
differences  among  users. 


yes  no  n/a 

—  -  1.  A  page-back  capability  allows  the  student  to  review 

previous  material. 

—  -  2.  Students  can  easily  exit  lessons,  return  to  menus,  and  exit 

the  program. 

—  —  —  3.  Student  has  control  over  the  rate  of  presentation  of  frames. 

—  -  4.  Flexibility  in  recognizing  correct  responses 

(e.g.,  misspellings,  partial  answers). 

—  —  —  5.  The  student  can  request  more  lengthy  messages,  through  helps, 

if  further  clarification  is  needed. 

—  —  —  6.  The  program  contains  activities  for  diagnosis  of  skills 

already  mastered. 

—  -  7.  There  are  remedial  exercises  for  skill  deficiencies. 

—  -  8.  Modularized  program  (with  menus)  allows  the  student  to  begin 

at  a  point  appropriate  to  past  achievement. 

—  —  —  9.  The  student  can  choose  the  difficulty  level  of  prdblems  or 

exercises  (achieved  by  variation  in  the  use  of  prcnpts). 

—  —  —  10.  The  student  can  correct  an  input  error  (e.g.,  with  BACKSPACE) 

or  recover  f ran  input  errors  without  disrupting  the  lesson 
sequence . 
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RESPONSIVENESS 


yes  no  n/a 


Feedback  is  informative  and 
timing  is  optimal. 


1.  When  the  student  must  stand  by,  periodic  feedback  indicates 
normal  operation. 

2.  The  ccnputer  tracks  response  patterns  and  lists  areas  where 
the  student  needs  remediation. 

3.  Feedback  and  directions  are  clearly  distinguishable  from 
other  text  through  use  of  color,  boxing,  reverse  video,  etc. 

4.  Students  can  obtain  a  score. 


5.  At  higher  mastery  levels,  students  are  given  immediate 
knowledge  of  right  and  wrong  responses,  and  more  lengthy 
feedback  is  delayed  until  the  end  of  the  session. 

6.  There  is  a  pause  after  feedback,  before  the  lesson  continues, 
to  allow  time  for  consolidation  of  the  newly  acquired  material. 

7.  Access  to  helps,  references,  or  resources  is  easily  available. 

8.  At  the  beginning  of  training,  feedback  is  response  informative 

(e.g.,  the - part  of  your  answer  is  incorrect.") 

9.  The  student  gets  more  them  one  chance  to  give  the  correct 
answer  (with  prompts). 

10.  It  takes  no  more  than  5  seconds  for  text  and  graphics  to 
fill  the  screen. 


NAVTRASYSCEH  TR86-002 


SCORING 

A  score  on  each  factor  (brevity,  consistency,  flexibility,  responsiveness  and 
compatibility)  can  be  detained.  To  dbtain  a  score  on  a  factor: 

1.  Add  the  total  nunber  of  "Yes"  responses. 

2.  Subtract  the  nunber  of  "N/A"  responses  frcm  the  total. 

3.  Divide  to  obtain  a  percentage  of  "Yes"  responses,  and  multiply  by  100%. 

Example:  Brevity  Score  =  Nunber  of  "Yes"  responses 

-  X  100% 

9  -  "N/A"  responses 
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